Skip to content

chore(airflow): Bump to 3.3.1 - #1665

Open
adwk67 wants to merge 9 commits into
mainfrom
chore/26.11/version-bumps/airflow
Open

chore(airflow): Bump to 3.3.1#1665
adwk67 wants to merge 9 commits into
mainfrom
chore/26.11/version-bumps/airflow

Conversation

@adwk67

@adwk67 adwk67 commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

Part of #1645

N.B.

  • most of the changes in this PR are due to updating the uv lock files
  • bump to node 26.9.0 not done possible until Nexus limits allow uploads again as it brings a new dependency with it - libatomic - which would need to be added to the image. Nothing requries node 26 so have left this at version 24.19.0.
  • bump to uv-version 0.12.15 not possible until action has been fixed:

jq: error (at sbom.json:64): Cannot iterate over null (null)
Possible fix:

-        jq '.components |= map(
  +        jq '.components |= ((. // []) | map(
                  if .properties then
                    ...
  -              | if .properties == [] then del(.properties) else . end)' \
  +              | if .properties == [] then del(.properties) else . end))' \
              sbom.json > sbom.slim.json

Important

Previously the opa-auth-manager was tested against one hard-coded version of airflow and the FAB provider, set in pyproject.toml. This was true for both in-image testing and local testing. For the airflow 3 tree that pin had drifted off every version we actually ship, so the tests were passing against an API the images no longer used. Now the specific version of airflow is tracked in the image and the auth-manager is tested against THAT version, as well as components that are specified in the constraints list and elsewhere (such as python). This means we are actually testing what we bake into the image. The builder stage runs once per image, so this happens for EVERY supported version - 2.9.3, 3.0.6, 3.2.2 and 3.3.1 - each against its own airflow, FAB provider and python. Running the tests by hand is the exception: there is no constraints file involved there, so the versions have to be named explicitly. That is what the local dependency group is for, and it is pinned to the latest supported version.

OPA tests failed due to mismatch between FAB provider and the opa-auth-manager. Fixed as part of this PR:

--- PASS: kuttl (710.00s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/opa_airflow-3.2.2_opa-latest-1.16.2_openshift-false (256.60s)
        --- PASS: kuttl/harness/opa_airflow-2.9.3_opa-latest-1.16.2_openshift-false (262.80s)
        --- PASS: kuttl/harness/opa-interop_airflow-latest-3.3.1_opa-latest-1.16.2_openshift-false (139.12s)
        --- PASS: kuttl/harness/opa_airflow-3.1.6_opa-latest-1.16.2_openshift-false (290.97s)
        --- PASS: kuttl/harness/opa_airflow-3.3.1_opa-latest-1.16.2_openshift-false (206.88s)
        --- PASS: kuttl/harness/opa_airflow-3.0.6_opa-latest-1.16.2_openshift-false (156.22s)
PASS

Nightly tests tested with stackabletech/airflow-operator#865:

--- PASS: kuttl (4389.48s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-3.3.1_openshift-false_executor-celery (266.78s)
        --- PASS: kuttl/harness/ca-cert_airflow-latest-3.3.1_openshift-false (296.60s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.3.1_ldap-authentication-server-verification-tls_openshift-false_executor-kubernetes (244.06s)
        --- PASS: kuttl/harness/remote-logging_airflow-latest-3.3.1_openshift-false_executor-kubernetes (353.92s)
        --- PASS: kuttl/harness/remote-logging_airflow-latest-3.3.1_openshift-false_executor-celery (248.43s)
        --- PASS: kuttl/harness/opa_airflow-3.3.1_opa-latest-1.16.2_openshift-false (229.79s)
        --- PASS: kuttl/harness/smoke_airflow-3.3.1_openshift-false_executor-kubernetes (185.52s)
        --- PASS: kuttl/harness/smoke_airflow-3.3.1_openshift-false_executor-celery (233.50s)
        --- PASS: kuttl/harness/oidc_airflow-latest-3.3.1_openshift-false (375.37s)
        --- PASS: kuttl/harness/versioning_airflow-latest-3.3.1_openshift-false (693.77s)
        --- PASS: kuttl/harness/triggerer_airflow-latest-3.3.1_openshift-false_executor-kubernetes (300.04s)
        --- PASS: kuttl/harness/triggerer_airflow-latest-3.3.1_openshift-false_executor-celery (210.85s)
        --- PASS: kuttl/harness/overrides_airflow-latest-3.3.1_openshift-false (185.54s)
        --- PASS: kuttl/harness/cluster-operation_airflow-latest-3.3.1_openshift-false (456.26s)
        --- PASS: kuttl/harness/orphaned-resources_airflow-latest-3.3.1_openshift-false (332.83s)
        --- PASS: kuttl/harness/external-access_airflow-3.3.1_openshift-false_executor-kubernetes (149.84s)
        --- PASS: kuttl/harness/external-access_airflow-3.3.1_openshift-false_executor-celery (172.54s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-3.3.1_openshift-false_executor-kubernetes_access-ssh (268.43s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-3.3.1_openshift-false_executor-kubernetes_access-https (269.07s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-3.3.1_openshift-false_executor-celery_access-ssh (330.65s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-3.3.1_openshift-false_executor-celery_access-https (223.79s)
        --- PASS: kuttl/harness/opa-interop_airflow-latest-3.3.1_opa-latest-1.16.2_openshift-false (141.39s)
        --- PASS: kuttl/harness/logging_airflow-3.3.1_openshift-false_executor-celery (393.13s)
        --- PASS: kuttl/harness/logging_airflow-3.3.1_openshift-false_executor-kubernetes (707.55s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.3.1_ldap-authentication-insecure-tls_openshift-false_executor-kubernetes (230.14s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.3.1_ldap-authentication-no-tls_openshift-false_executor-kubernetes (203.24s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.3.1_ldap-authentication-server-verification-tls_openshift-false_executor-celery (231.53s)
        --- PASS: kuttl/harness/resources_airflow-latest-3.3.1_openshift-false (168.21s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.3.1_ldap-authentication-no-tls_openshift-false_executor-celery (243.93s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.3.1_ldap-authentication-insecure-tls_openshift-false_executor-celery (232.98s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-3.3.1_openshift-false_executor-kubernetes (183.23s)
PASS

Definition of Done Checklist

Note

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.

Please make sure all these things are done and tick the boxes

  • Changes are OpenShift compatible
  • All added packages (via microdnf or otherwise) have a comment on why they are added
  • Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
  • All packages should have (if available) signatures/hashes verified
  • Add an entry to the CHANGELOG.md file
  • Integration tests ran successfully
TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

boil build <IMAGE> --image-version <RELEASE_VERSION> --strip-architecture --load
kind load docker-image <MANIFEST_URI> --name=<name-of-your-test-cluster>

See the output of boil to retrieve the image manifest URI for <MANIFEST_URI>.

@adwk67 adwk67 mentioned this pull request Sep 17, 2026
Open
11 tasks
@adwk67 adwk67 self-assigned this Sep 17, 2026
@adwk67
adwk67 marked this pull request as ready for review September 17, 2026 11:38
@adwk67 adwk67 mentioned this pull request Sep 17, 2026
Merged
6 tasks
@adwk67
adwk67 marked this pull request as draft September 17, 2026 13:40
@adwk67 adwk67 moved this to Development: In Progress in Stackable Engineering Sep 17, 2026
@adwk67 adwk67 mentioned this pull request Sep 18, 2026
Open
20 tasks
@adwk67
adwk67 marked this pull request as ready for review September 18, 2026 12:02
@adwk67 adwk67 moved this from Development: In Progress to Development: Waiting for Review in Stackable Engineering Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@adwk67 adwk67

Labels

scheduled-for/26.11.0

Projects

Stackable Engineering
Status: Development: Waiting for Review

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@adwk67